youtube
Here's the thing nobody tells you about local AI speed: for every single token a model generates, it rereads itself entirely from memory. All 17GB, every token. That's why memory bandwidth sets a hard speed ceiling, and the formula is simple: bandwidth divided by model size.
On an M5 Max at 614 GB/s, a 17GB model predicts around 36 tokens per second, with real benchmarks closer to 31. It's not exact, but it gets you close to the max speed you'll hit before any optimization, useful to know before you buy.
#techwithtim #ai #machinelearning
|
2026/10/20
youtube
